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ABSTRACT 



Textbooks used for the first time in a large 
introductory, and in an advanced, psychology course were 
systematically examined, using different methods for each text, to 
measure the amount of individual student underlining. This amount was 
then correlated, in each course, to the students 1 performance, with 
each student’s general academic ability being considered as a control 
variable in the introductory course. Results indicate: (1) a high 

degree of reliability in measuring the amount of underlining was 
obtained, using a small number of randomly related pages of a text; 

(2) underlining was used more frequently by students of lower 
academic ability; and (3) it was not clear that underlining 
facilitated achievement. It is concluded that underlining would not 
serve too well as an index of identification of relevant content due 
to the large proportion of students (about 50%) who did not underline 
to any measurable extent in their text. (Author/TL) 
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This project was carried out to obtain some preliminary data * * 
concerning the me t 'nods and approaches that might be used to investi- 
gate questions pertaining to student identification of relevant in- 
structional content * 

Among t:.e student connected factors likely to affect achieve- 
ment in college level courses is the adequacy with which the student 
identifies the most relevant instructional content among the to tad. 
instructional input available in lecture and assigned textbook 
materials* 



Variation in student course achievement would seem necessarily 
to relate quite highly to how well the central points, items of 
knowledge and information being given primary emphasis in the 
measures of achievement were identified as such within the instruc- 
tional materials by the student* This factor, the identification 
of relevant content (IRC), would seem to operate in addition to how 
well subject nu;tter content was comprehended or understood since 
the latter will effect performance only if it occurs for the most 
relevant content* 

Several recent studies have provided evidence indicating that 
IRC is a factor related to acquisition of information contained in 
textual and instructional materials* Rothkopf and Bisbicos ( b ), 
Frase (3)1 and Bruning ( 1 ) have shown that the frequency and 
content of test items following sections of textual material reliably 
affects the type and amount of material learned* This presumably 
is due to differential attention to certain of the material as a 
result of the specific content focus of the test items. Other studies 
reviewed by Gagne ( 4 ) indicate that providing explicit IRC with 
instructional materials rather than relying on the learner’s IRC 
sometimes facilitates and sometimes retards acquisition of the 
specified material. Carrol ( 2 ) has commented generally on the 
limited evidence and research concerning the nature of the information 
processing skills and strategies employed in the acquisition of 
knowledge from written materials* 

Although a number of instructional conditions no doubt affect 
IRC, this factor is probably of more consequence in general survey 
courses having large enrollments, few examinations and little in- 
structor-student contact which can provide the best basis for 
successful IRC* 

Determination of the extent and variation in successful IRC 
among students and the conditions affecting it should contribute 
to improvement in college level instruction and also provide a 
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v/r.icn provide for the lending and return of texts and identifi- 
cation of die user, it appeared feasible to examine some character- 
istics of textbook underlining to see whether they might provide a basi^ 
for subsequent investigation of I3C in learning at the college level. 
The specific purpose of the project was then to make some assessment 
of the extent and nature of textbook underlining as r possible index 
of that ISC activity moot likely to be directly reflected in actual 
course achievement* 



The objectives of the project were to determine: 

1. The amount of underlining occurring in an introductory survey 
course texts and in advanced upper division course texts. 

2. The reliability of several methods used to .quantify the amount 
of underlining that occurs. 

3. Whether there was any general relationship between amount of 
underlining and students 1 individual course achievement* 

Because of the extensive labor that might be required to quantify 
the amount and other characteristics of textbook underlining, several 
alternate methods were used. Cne of these was a method which was ex- 
pected to lack precision but would permit in a limited amount of time 
consideration of a large number of texts and thereby give a better 
picture of the possible variations in underlining practices. The 
other represented a more precise method appropriate for a small number 
cf texts in which there was likely to be more extensive underlining. 

It was expected that a positive relationship should exist between 
amount of underlining and level of course achievement. Underlining 
is an overt indication of some amount of selective processing of text- 
book content and thereby would suggest greater attention to and in- 
volvement. with the material - a condition essential to learning. 

Amount of underlining, as such, however, would not indicate that the 
most relevant content was being identified in this way. 



METHOD 



To investigate these questions, observations were made concerning 
the underlining practices followed by students in textbooks used in an 
introductory and an advanced psychology course, the texts having been 
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borrowed from and return. a to the textbook library* 

For each student having a text included in the observations, 
performance over the semester in the respective course was de- 
termined. For the majority of students in the introductory 
course, college entrance test, scores were also obtained as an 
estimate of their general academic ability. 

Introductory Psycholo ~j course : Judgments were made con- 

cerning the amount and nature of underlining (or highlighting) 
that appeared in an available sample of 24? textbooks which 
had been returned to the textbook library after use for one 
semester by individual students in each of four sections of 
Introductory (General) Psychology. The judgments were made by 
a student assistant who was instructed to systematically page 
through each text and to estimate the proportion of pages for 
which there was respectively a light, moderate and heavy amount 
of underlining. A reference point for this judgment was provided 
by sample pages of text which had been underlined to indicate 
the upper limits of the light and moderate judgment categories, 
respectively. The lower limit for the ’’light underlining 11 
category was at least one word underlined. This procedure also 
provided an estimate of the proportion of pages having no under- 
lining. The jucge was further instructed to attend to the style 
of underlining employed; that is, whether the student had mainly 
underlined words, phrases, sentences or paragraphs or combina- 
tions of these in those texts in which underlining was estimated 
to occur on 10 percent or more of the pages. The judgments and 
observations for each text as it was reviewed were recorded on a 
standard form. 

To obtain some indication of the reliability of the estima- 
tion procedure for indexing the amount of underlining, a second 
student assistant made the same judgments independently for a 
random subsarnple of the same textbooks. The degree of corres- 
pondence between the two independent judgments was th$n deter- 
mined. 



In each of the four sections, . the students total score on 
all tests administered during the semester was used as a measure 
of academic performance in that course. In all instances these 
were objective tests prepared by the class instructor. Since^ 
among the several introductory course sections, the number, con- 
tent, and length, etc. of the tests varied somewhat, it was not 
possible to equate these performance measures across sections. 
Consequently, the relations with average test performance was de- 
termined separately for each of the introductory sections* 
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Tc .'.o tormir.c the relations between amount of underlining and 
cou.'ae performance independent of (that is , controlling for) general 
academic ability, the composite score obtained on the ACT ( American 
Crg_j H Tea J ;. ) t which is a required entrance test, was gathered for 
as m-r.y students in the sample as possible from the student Record 
Center . 

Audit:, oeel Observations : In a separate subsequent Introductory 

Psychology class, a one-page ,f study habits" questionnaire was admin- 
istered which among other things, inquired about the students* use of 
textbook underlining in the course* The relevance of the questionnaire 
inquiry to this project was that reported amounts of underlining could 
then be compared in a general way to the underlining that had actually 
been observed to determine how valid the more readily obtained self- 
report information might be. 

Advanced Psych olorv Course: For a smaller, uiroer division 

psychology class (Psychological Tests and Measurements) using a new 
text for the first time, the book used by each member was examined 
with respect to the underlining practices of the student. In each chapter 
that had been assigned during the semester two 2-3 page sections were 
selected from among those havirig predominantly • factual content which 
showed within the section variation with respect to degree of specif- 
icity. These sections were those that appeared to contain material 
which students would be likely to identify as having more or less 
specific achievement relevant items of knowledge or information. 

An index of amount of underlining was then obtained in the follow- 
ing way for each text having at least some underlining. In each of 
the sections selected the number of lines of print having one or more words 
underlined were counted along with the total number of lines of print 
in the section. For each student's text the total number and propor- 
tion of total possible lines that could be underlined was determined. 

A separate index of the amount of individual student underlining 
was also obtained by applying the same "line-count" procedure to eleven 
additional randomly selected pages in the text. The latter pages 
were selected from among those in the alternate chapters that had been 
assigned for the course. Again for each student the total number of 
lines with underlining was determined. This index was obtained to 
determine the adequacy of a random as opposed to a selected page index 
of amount of underlining. 

The reliability of the index of the amount of underlining said 
for the text in this course was provided by correlating the total 
number of lines obtained for each individual for separate randomly 
designated halves of the textbook sections in which these values were 
obtained. This procedure is analogous to the split-halves reliability 
method used to assess the internal consistency among items in achieve- 
ment tests. To use this method however, texts containing no underlining 
at all could not be included since this characteristic necessarily re- 



r-uired maximum consistency in this respect through the whole text, and 
would have unrealistically inflated the reliability estimate. 

ns in the introductory course, achievement was measured by the 
students' performance through the semester on instructor-made 
objective tests. This achievement criterion did not include course 
required creative applications such as term papers or test evaluations. 

RESULTS 

Introductory Psychology Course : Table I shows the number and percent 

of students at each estimated level of proportion of pa£es underlined 
a slijht amount .or more for students in separate and • combined sections 
of Introductory Psychology. It is evident that in the introductory 
course students did not utilise text underlining to any great extent. 
Slightly more than one-half of the sample of texts did not reveal any 
underlining at all and only about 20 percent were estimated to have 
underlined material on more than 15 percent of the pages. It does 
appear that there is a certain amount of variation in this respect 
between the separate sections of this course. Tp determine whether 
the section differences represented chance variation, a Chi-square sest 
was applied to the frequencies for the none, 1-15 an ^ 16 plus, percent 
underlined categories. A Chi-square value of 13.^, with 6 d.f.was 
obtained which is significant at the .05 level, indicating that the 
section differences may reflect other than chance factors. 

i 

Table II shows the number of texts, among those for which there 
was some underlining, having each designated proportion of slight, 
moderate and heavy amounts of underlining. 

It appears that slight and moderate amounts of underlining 
occurred for about the same proportion of students but that moderate 
underlining occurred on a greater proportion of pages than did 
slight underlining. The latter tendency also exists for heavy 
relative to slight underlining. In the texts in which slight under- 
lining appears it does so on a smaller proportion of pages than 
does moderate or heavy underlining. 

With respect to the style of underlining used by those estimated 
to have 10 percent or more of the pages underlined, only 20 percent 
were found to have used either words, phrases, sentences or paragraphs 
predominantly. For the remainder some combination of these was most 
evident with words, phrases and sentences being most frequent combina- 
tion, accounting for 21 percent of the total. Table III shows the 
frequency and percent occurrence among students pf each of these four 



ViJur.cer and percent of students at each level of 
estimated percent of pn^es underlined for each section 
in introductory psychology. 
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Numbers of students . ■ t e;.ch c* t in\a ted lcv.il of 
occurrence of different amounts of underlining 
for all introductory psychology sections combined 
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Table 3 

Number and proportion of students 
exhibiting each type of underlining 
in Introductory Psychology texts. 

Tyne Number c /o of total* 



words 


27 


44 
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paragraphs 
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either in combination or separately • The greatest 
underlined on the basis of whole sentences. 



Table TV provides a cross-tabulation of the estimated proportions of 
textbook pages with underlining made by each of the two judges for the 
subset of 111 texts for which both judgments were made. An index of the 
degree of agreement between judgments was obtained using the statistic 
suggested by hays ( 5 ) which gives the exact probability of two randomly 
drawn pairs of observations having the same relative order for both 
judgments. A value of .91 was obtained for' when these classification 
categories were used. Tor these categories, there was agreement on 63 
percent of the cases. It is evident that the greatest amount of agreement 
occurred for texts that apparently were net underlined at all and that 
there was no a Vv*ry high degree of agreement using this method on texts 
for which some amount of underlining had been done. Because of the nature 
of the distribution of values for the percent-cf-pages underlined as 
shown above, i.e M a large proportion of students showing no underlining 
and a non-normal distribution across the range of percentage values, this 
index was treated in this analysis as an ordered categorical variable rather 
than a continuous variable. Students in each introductory section was 
cross-classified with respect to ACT score level (3 levels) and estimated 
percont-cf-pages underlined, i.e., none, 1-5 percent, and 6 percent or 
greater - as shown in Table V. Within each of the cross classification 
cells, the students* average class test score was determined. The weighted 
marginal means for the ACT and underlining percent categories and the overall 
(•weighted) means v/ere also determined and are shown with the cell means in 
Table VI. Because of the larger number of texts for which this judgment was 
obtained, the first judges’ estimates of amount of underlining (rather than 
the second judges*) v/ere used for the analysis shown in Tables V and VI. A 
similar analysis was made using the underlining estimates given by the second 
judge and showed quite similar trends. 



lamination of Table V shows a tendency in all sections, except section 
for a larger proportion of lower than of higher academic ability students to 
engage in some amount of underlining. There does not appear to be any 
differential trend in this respect in Section 2. Combining all sections and 
comparing frequencies for none (zero percent) with some underlining (i.e., 1 
pr-vennt or greater) for the three ACT levels yields a'y 2 value of 12.26 
(with 2 d.f„) wh.ich is significant at the .005 probability level. 



examination of the cell and marginal means shoym in Table 31 does not 
reveal a strongly consistent tendency for those who had engaged in more under- 
lining to achieve at a higher level than those who had not. At the same time, 
the trend in general js more in this direction (i.e. f toward a better per- 
formance for those showing more underlining) than in the opposite direction. 
This tendency is most evident in Sectiqn 2. 
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cause of the small frequencies in the colls and the 
ty of treating each section oeparn tely , there is no 
leal test which couiJ bo appropriately applied to this 
determine the statistical reliability of any overall 
or differences with respect to the measure of course 
ar.ee . 



.Advanced Psy 
of underlining in 
pcrcent-cf-lines 
tier* of Table VII 
was nc underlinin 
than one-third of 
6 percent or more 



choloyy Course The distribution of amount 
the advanced course text in terms of the 
on selected pages is shown on the lower par- 
. For one-half of the cl:; ss members there 
g on any of the pages, while slightly more 
the class showed underlined words, etc*, on 
of the lines in the selected sections* 



Considering the 9 randomly selected pages v 9 (36/j) of the 
26 class members had done some underlining cn JO percent or 
more of the pages# It ap] e,;rs that when underlining occurred, 
it occurred much more extensively in the advanced than in the 
introductory course texts. 



For the advanced psychology course text, each of the two 
sections selected in each of the 16 chapters were randomly 
assigned to one of two categories. The number of lines con- 
taining underlining in all sections in each of the two categ- 
ories we re determined for each text and these two values 
correlated across texts for the 13 students who did some under- 
lining. A Spearman Rank-order correlation coefficient was com- 
puted because of the small N and non-normality of the distribution* 
The value obtained for the coefficient was *95* A similar 
procedure applied to values for this variable obtained in the 
same way for the randomly selected pages yielded a rank-order: 
coefficient of . 83 . The rank-order correlation betv/een the 
values for amount of underlining in the selected sections and 
the random pages was .93 which ihdicates that the value obtained 
from 9 pages gives the same relative ordering as that obtained 
from more than 35 pages. 



It is evident that the index of underlining obtained by 
counting the number of lines of print with at least minimal 
underlining is a fairly consistent relative index of amount of 
underlining. It should be noted, however, that the split- 
halves technique used here was carried out so that sections 
or pages at all points through the text were represented equally 
often in both of the halves being correlated. This was, done 
to minimize the possibility that a student's shift in under- 
lining practices from the beginning to the end of the text would 
reduce the magnitude of the coefficient, i*.e#, to estimate the 
reliability of the overall index. 
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Classification of a<?v: need psycnoxogy 
0 i..«r, students with respect to percent 
o?' li-es in selected text sections con- 
taining under! ini ng and overall class 
test performance . 
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To determine hew consistent students wore in their underlining 
■:r c:ices through the whole text , rank-order correlations were also 
c\ Ir.iuod between the amount of underlining in successive thirds of 
tl.j selected sections. The coefficients obtained for the underlining 
frequency between sections in the first and second thirds of the text, 
t'/.e second and third thirds and the first and third thirds, respectively, 
.3 o, .3-4, and .64. It appears that there is some shift in the relative 
magnitude of the values obtained for this index as the student moves 
through the book and the course and that the amount of underlining 
represented by this index tends tc be more stable near the end than 
the beginning of the text (and course). 



To determine if there was any correspondence between amount of 
underlining and course achievement , students were cross classified 
with respect to category levels of each of these variables. These 
results are shown in Table VII. From inspection of the table, it is 
evident that if anything there is a tendency for those who performed 
best to be less likely to have dene any underlining. Those who did 
the- moot underlining appeared equally frequently at all three perform- 
ance levels. 



Additional Observations : On a questionnaire administered to 

students in a subsequent introductory course, better than 70 percent 
indicated that they had underlined- material in their texts at least 
"occasionally" • This is a significantly greater proportion than were 
found to have underlined by actually checking texts of previous 
c lasses . 



CONCLUSIONS AND RECOMMENDATIONS 

One of the more apparent observations concerning underlining as an 
index of successful student behavior is that only about 50 percent of 
the .students in either the introductory or advanced courses were found 
to engage in textbook underlining. At the same time, however, a much 
larger proportion of students reported, when asked, that they did in 
fact do some textbook underlining. The latter observation indicates 
that questionnaire indices concerning this means of IRC could at best 
provide a relative, but neither an absolute nor a very precise measure 
of the amount of or other characteristics of underlining, nor one 
likely to have very high validity. 

Although the same proportion of students exhibited no underlining 
in the introductory as the advanced course, those that did underline 
in the* advanced course did so to a much greater extent than those in the 
introductory course. It is possible that students who adopt this 
practice tend to use it more extensively as their educational career 
progresses. 




as che student goes through the text. The practices followed tend 
to be more stable nearer the end than the beginning; of the semester. 



of the greater reliability when the value for the text as a whole 
is considered. 

In both the introductory and advanced courses, it was quite 



In fact, there was a tendency for students having lower general 
academic proficiency to engage in underlining more than those with 
hi.jher general proficiency. This observation is consistent with 
those showing low negative relations between academic ability and 
time spent studying which seems to reflect the greater effort re- 
quired of those having less ability. However, when controlling for 
general academic ability, the data did not consistently show that 
underlining contributed to better achievement although the results 
te-r.ded more in this direction than otherwise. The latl er tendency, 
however, may reflect the effects of motivational or effort factors as 
much as any direct facilitating effects of underlining, as such. 

In general, it appears that actual textbook underlining as a 
means of indexing student IRC is quite limited. This is primarily 
because of the large proportion of students who do not employ 
underlining as a means of IRC. Results further suggest that under- 
lining is not used to the same extent by students of different 
academic proficiency levels and that motivational factors may also be 
involved in relations between underlining and performance in actual 
courses. 

All in all, it would seem that to the investigator of student 
IRC as a factor in knowledge acquisition might better be conducted 
under more controlled conditions using strictly experimental rather 
than natural observation or correlational methods. 

At the same time given the moderately high consistency of under- 
lining occurrence textbook observations might be a fairly good means 
of determining the qualitative variations that exist for this means , 



The latter observation raises some question as to the significance 



apparent from the successful performance of some of those who did 
net underline that this practice is not the only procedure for IRC 
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